Light modulation remover

ABSTRACT

The quality of recorded images with for example a high speed camera can be detonated by light modulation which is a consequence of for example gas-discharge lamps running at a lower frequency with respect to the camera.  
     A camera according to the invention provides light modulation removal means which tackle this problem by handling fields of the image different depending on the light modulation.

[0001] The invention relates to a camera as described in the preamble ofclaim 1.

[0002] The invention further relates to a method of recording an image.

[0003] These cameras are known and are used for example to broadcastsport events etcetera. In such a case a camera can be used that runs atfor example three times the normal picture rate (150/180 Hz). When suchcamera has to operate for example under artificial light, although ingeneral much effort is paid to good light conditions, the camera cansuffer from severe unwanted light modulation effects working under forexample gas-discharge lamps.

[0004] A disadvantage of a camera working under such light conditions isthat the 50/60 Hz beat frequency between the 150/180 Hz camera and therectified-mains components in the light source (100/120 Hz) generates anunwanted modulation of the video signal. Even though the beat frequencyitself is already visible, the recorded video is supposed to be viewedat normal speed, which down converts the beat frequency by a factor 3(when the camera runs at three times the normal picture rate). When thecamera has to operate under gas-discharge lamps the modulation can geteven worse. To improve this situation it is proposed to spread the lampsover all the three available main-faces, but this still does not solvethe problem. This is caused, for example because some objects are lessilluminated or due to some reason reflect only one or two light faces.In consequence, a complex unwanted light modulation of the images is theresult. An additional problem is the light changing color temperatureduring the cyclic discharge of the lamp.

[0005] It is inter alia an object of the invention to provide a cameraand a method, which does not have the disadvantages of the prior artcamera. It is further an object to provide light modulation removalmeans for use in such a camera system.

[0006] This is achieved according to the invention by a camera asdescribed in claim 1. The solution is proposed by the camera accordingto the invention, which is based on the processing of the images duringthe least common multiple of the camera acquisition and the rectifiedmains-period. This beat frequency period separates images having thesame light modulation, the only difference here are the effects ofmovement on the scene. An embodiment of the invention is described inthe dependent Claims.

[0007] These and other objects of the invention will be elucidated withreference to the Figures.

[0008]FIG. 1 shows a schematic embodiment of a camera according to theinvention,

[0009]FIG. 2 shows an embodiment of a processing unit according to theinvention.

[0010]FIG. 1 shows a schematic embodiment of a camera CM1 according tothe invention. The camera comprises an image sensor IS1 which suppliesRGB signals to a sensor unit SU1. The sensor unit is coupled to aprocessing unit PU1 for processing the received RGB signals andsupplying YCrCb signals to a light modulation removal unit LMRU1 whichis coupled to an end processing unit EPU1. The output of the endprocessing unit supplies the signals which are known by the man skilledin the art and for example are coupled to a recording device, etc.. Thelight modulation removal unit LMRU1 will be described in more detailwith reference to FIG. 2 hereafter.

[0011]FIG. 2 shows in more detail block schematic a light modulationremoval unit LMRU2 according to the invention. At a video input VI thelight modulation removal unit receives the YCrCb signals from theprocessing unit PU2 (see FIG. 1). The light modulation removal unitLMRU2 comprises different parts. A first part having in this examplefive field memories FM1-FM5 for storing different fields of the receivedsignal. A second part comprising of (in this example) threede-interlacers DIL1-DIL3. Further a part having three motion detectorsMD1-MD3. Further a part having a modulation estimator ME and a parthaving an averager A. After the different fields are stored in the fieldmemories FM1-FM5 the fields are supplied to three de-interlacers wherebythe first de-interlacer DIL1 receives at a first input the input signalsafter filtered in a band split filter BF1 and at a second input theoutput signals of FM3. The second de-interlacer DIL2 receives at a firstinput the output signals of FM1 after filtering in a second band splitfilter BF2 and at a second input receives the output signal of a fieldmemory FM4. The third de-interlacer DIL3 receives at the first inputsignals at output of field memory FM2 and at the second input the outputsignals after field memory FM5. The output of the de-interlacersDIL1-DIL3 is supplied as a first input to respectively motion detectorMD1-MD3. At the second input the motion detector MD1 receives the outputsignal of field memory FM3. At a second input the motion detector MD2receives the output signal of FM4 and the second input of motiondetector MD3 receives the output signal of field memory FM5. The outputsof the motion detectors MD1-MD3 are supplied to a maximizer MX whichsupplies at the output the maximum of the three input signals. Thisoutput signal is supplied via an amplifier AMP to a subtractor at thenegative input. The amplifier AMP receives at a control input a motionsensitive signal MSS.

[0012] The modulation estimator ME receives at a first input the outputsignal of the de-interlacer DIL1, at a second input the output signal ofthe band split filter BF2 and at the third input the output signal ofthe de-interlacer DIL3. The output of the modulation estimator ME iscoupled via a unit LUT2 to an amplifier AMP2. At the control input theamplifier AMP2 receives the output signal of the subtractor SUB. Theoutput of the amplifier AMP2 is coupled via a third amplifier AMP3 to afader unit FU. The output of the fader unit is coupled via a summingunit SUM to the output VO of the light modulation removal unit LMRU2.

[0013] The averager A receives at the first input the output signal ofthe de-interlacer DIL1 at the second input the output signal of the bandsplit filter BF2 and at the third input the output of the de-interlacerDIL3. The averager gives at the output the average of these three inputsignals to the fader unit FU and also via a low-pass filter LPF and aunit LUT1 to the amplifier AMP3 as a control signal.

[0014] In this way it is possible to remove the light modulations of thereceived input signals. The solution is based on the processing of theimages during the smallest common product of the camera acquisition andthe rectified-mains period. This beat frequency period separates imageshaving the same light modulation; the only differences here are theeffect of movement on the scene. Further, for stationary scenes, anaverage of the images during the beat frequency period gives a fullremoval of light modulation as a result.

[0015] As temporal averaging of pictures result in movement blur, amovement detector is used to fade the output between averages andoriginal video. The movement detection is based on differences betweenfields with the same light modulation. A low-pass filter, to reducedecision noise, filters the max of differences of all light phases.Before feeding the motion control to the fader, a linear gain realizes asensitiveness adjustment. The beat frequency is assumed equal to thestandard field repetition divided by the highest common factor of fieldfrequency and light frequency e.g. 150 Hz camera with light frequency100 Hz (2x the main frequency)=150/HCF(150,100)=3 different illuminationphases for acquisition. Having a 50 Hz camera with light frequency of120 HZ, this results in 50/HCF(120,50)=5 fields.

[0016] In case the original images are available in interlaced format,consecutive picture can not just be averaged together without a seriousreduction in vertical resolution of the respective video material. Usingconsecutive images of an interlaced video stream to extract imagefeatures like motion and others is also very difficult in images areascontaining high vertical spatial frequencies. The reason for thoseproblems is that two consecutive fields do not represent the samespatial position of the image. Therefore de-interlacing techniques havebeen used to restore the complete frame at each temporal position of theoriginal image fields. In this way all spatial positions of the imagesare available for any processing at any needed input field time. Toprevent the unwanted modulation, present on the original video images,of disturbing the de-interlacing process, the inputs are taken with atemporal distance equal to the said common period, in this way nomodulation differences between both images is expected to be present.

[0017] Looking to stationary pictures, the quality of the averagedoutput depends on the used de-interlacer. In the case of a progressiveinput, the de-interlacer becomes redundant. Here averaging gives also awanted noise reduction effect. For moving objects in the picture, thelight modulation is still present. Although, in general, this concernsonly small portions of the total disturbances, depending on the scene,their presence is still annoying. Up on that, the visibility of thisresidual light modulation is emphasized by the absence of the removedoverall disturbances.

[0018] Moving areas with small details, where motion detection fails,are also averaged, leading to detail loss (e.g. grass during camerapanning). To avoid this, estimation is made of the local strength of thelight modulation. If no modulation is measured at the input of theaverage function the fader is set towards the original video.

[0019] To further reduce artifacts the averaging action is also madeinversely proportional to the local luminance value. This can beintroduced in the system due to the light flicker sensitivity of thehuman eye. Tempering the averaging action where the light modulation isalready less visible.

[0020] A big improvement can be reached by applying motion compensationtechniques, interpolating pictures from different light phases to thesame time moment before averaging them together.

1. Camera for recording pictures comprising an image sensor forreceiving a picture, a processing unit for processing the picture and anend processing unit, characterized in that the camera comprises a lightmodulation removal means between the processing unit and the endprocessing unit for removing light modulation between different fieldsof the picture.
 2. Camera as claimed in claim 1 , characterized in thatthe light modulation removal means comprise adaptive fading means forfading between one field and at least n fields, whereby n is therepetition pattern of light modulation.
 3. Camera as claimed in claim 2, characterized in that the light modulation removal means comprisemeans to calculate the lowest common multiple of the repetition periodof said illumination variation and the repetition period of saidpicture, which lowest common multiple is used as common period toaverage consecutive images of said picture during recording.
 4. Cameraas claimed in claim 3 , characterized in that the light modulationremoval means comprise a motion detector and means to decrease theaveraging of consecutive images when motion is detected, which motiondetector comprises evaluation means to evaluate the local differencebetween images having a field difference of n.
 5. Camera as claimed inclaim 3 , characterized in that the light modulation removal meanscomprises means to estimate the modulation strength on a locality of theimage, and reducing means to reduce the averaging on localities wherethe light modulation is weak.
 6. Camera as claimed in claim 3 ,characterized in that the light modulation means comprises means toreduce the averaging on localities where the luminance component of saidpicture is low.
 7. Camera as claimed in claim 3 , characterized in thatmeans to exclude high spatial frequency components of the picture fromthe averaging step.
 8. Camera as claimed in claim 3 , characterized inthat the light modulation removal means comprise means to correctconsecutive images to the same temporal position using motioncompensated conversion techniques prior to the averaging.
 9. Camera asclaimed in claim 1 , characterized in that the light modulation removalmeans comprise de-interlacing means to generated information fro anymissing position in the original interlaced image, using two images withdifferent interlace phases and equal light modulation phases.
 10. Lightmodulation removal means for use in a camera system according to claim
 1. 11. Method of removing light modulation during recording pictures withan image sensor having the step of receiving the picture, processing thepicture, removing the light modulation by storing different field of thepicture and averaging the different fields in dependence of motion,and/or locations with low respectively high luminance locations.